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Executive summary 

The project 

Catch Up 1 Literacy is a structured one-to-one literacy intervention for pupils between the ages of 6 
and 14 who are struggling to learn to read. It teaches pupils to blend phonemes (combine letter 
sounds into words), segment phonemes (separate words into letter sounds), and memorise particular 
words so they can be understood without needing to use phonics strategies to decode them. The 
intervention matches books to pupils according to their reading ability, which pupils then read to a 
teaching assistant (TA), so is intended to also support the development of their comprehension skills. 

In this evaluation, the intervention was delivered through two 15-minute sessions per week over 30 
weeks at the transition from primary to secondary school, with a break for the summer holidays. 
Pupils were identified by their Year 6 teachers in their feeder primary schools as being struggling 
readers who were predicted to achieve below level 4b in reading. Each secondary school employed 
two part-time TAs to deliver the intervention in the last few weeks of Year 6 and after the pupils 
transitioned to secondary school. The TAs delivering Catch Up Literacy were supplied with detailed 
session plans and received three half-day training sessions led by Catch Up. 

The study was funded by the Education Endowment Foundation as one of 23 projects focused on 
literacy catch-up at the primary — secondary transition. 


Key conclusions 

1 . 

The pupils that received Catch Up Literacy made more progress than pupils that did not. 
However, this difference was not statistically significant so we cannot be confident that it was 
not due to chance. 

2. 

Catch Up Literacy did have a statistically significant impact on pupils’ attitudes to school, self- 
assessed ability in reading, and their confidence in and enjoyment of writing. 

3. 

Schools should ensure that Catch Up Literacy sessions are located in a private and quiet 
location, and that teaching assistants are given adequate time to prepare before each lesson. 

4. 

Teaching assistants reported a number of benefits for their own professional development. 
These include increases in confidence, knowledge of literacy support and overall job 
satisfaction. 

5. 

Future research could test the impact of Catch Up Literacy against an “active” control group 
that receives the same amount of one-to-one tuition as the pupils who receive Catch Up 
Literacy. 


What impact did it have? 

The pupils who took part in Catch Up Literacy made more progress than the pupils in the control 
group. This difference was not statistically significant, so we do not have sufficient evidence to 
confidently conclude that the effect did not occur by chance. The effects of the intervention on pupils 
of different genders and pupils who are eligible for free school meals (FSM) were similar to those in 
the main analysis, but also not statistically significant. 

Positive and statistically significant outcomes were recorded on other measures. Pupils who received 
the Catch Up intervention appeared to develop more positive attitudes towards school, a higher self- 
reported score for reading confidence and ability, and a more positive rating for writing confidence 
and enjoyment. These outcomes had effect sizes of 0.25, 0.32 and 0.23 respectively. The process 


1 Catch Up® is a not-for-profit UK registered charity (1072425). Catch Up® is a registered trademark. 
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evaluation indicated that most TAs believed Catch Up Literacy had a number of benefits for their own 
professional development, and a positive impact on pupils’ confidence and engagement with learning. 

How secure is this finding? 

The primary analysis is judged to be of moderate to high security and was awarded a security rating 
of 4 padlocks. The evaluation was set up as a randomised controlled trial. A maximum of 60 pupils 
from each of the 15 secondary schools were randomly assigned to one of three groups: a control 
group that received “business-as-usual” classroom teaching, an intervention group that received 
Catch Up Literacy, and a reserve group. Due to timing issues, pupils were selected prior to sitting 
their SATs. As Key Stage 2 results were used to control for prior attainment in the final analysis, the 
reserve group was used to replace pupils in the experimental groups if they did not complete Key 
Stage assessments. The developer led the training and oversaw the provision of the intervention, so it 
should be considered an efficacy trial. 

Attrition was just above 5%, but this was evenly spread across the control and intervention groups, 
and there was no evidence that the attrition led to a biased trial. Pupils’ literacy skills were estimated 
using the New Group Reading Test (Test 3a and 3b). Blind marking of test papers was undertaken, 
but both the attainment tests and the intervention were delivered by the same TAs and this should be 
considered as a threat to the validity of the measurements. 

Pupils in the intervention group received one-to-one tuition as part of Catch Up Literacy, whereas the 
control group received normal classroom teaching. This introduces the possibility that a proportion of 
any observed effect was caused by the more intensive tuition received by the intervention group, not 
by any particular characteristics of the Catch Up Literacy intervention. 

How much does it cost? 

All of the direct costs to schools were borne by the EEF grant for the project. The evaluator used this 
information and the number of pupils receiving the intervention to estimate that Catch Up Literacy 
cost an average of £769 per pupil in this evaluation. This estimate includes the salary costs of TAs 
and the cost of the training provided by Catch Up. This cost will be lower if more pupils are involved in 
the intervention, as the cost of training a TA will be spread out over more pupils. The cost will also be 
lower if schools do not implement Catch Up Literacy as an additional activity that requires paying TAs 
for more time, but use it to replace things they were already doing, as this will eliminate the salary 
cost from the cost estimate above. The training provided by Catch Up costs £350 per attendee. The 
independent evaluator did not look at additional costs that were associated with attendance at this 
training or additional resources that may have been required to deliver the intervention successfully. 


Group 

No. of pupils 

Effect size 

(95% confidence 
interval) 

Estimated 

months’ 

progress 

Evidence 

strength* 

Cost** 

Intervention vs 
control 

(all pupils) 

557 pupils 
(15 

schools) 

0.12 

(-0.02 - 0.25) 

+2 months 

A A A A 

££££ 

Intervention 
vs control 
(FSM) 

115 pupils 
(15 schools) 

0.004 

(-0.30-0.30) 

0 months 


££££ 

* For more information about evidence ratings, see Appendix C in the main evaluation report. Evidence ratings are not provided 
for sub-group analyses, which will always be less secure than overall findings. 

** For more information about cost ratings, see Appendix D in the main evaluation report. 
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Introduction 

Intervention 

Catch Up® Literacy is a structured one-to-one literacy intervention for learners from the age of 6 to 14 
years who are struggling to learn to read. The intervention adopts a combination of segmenting and 
blending phonemes, and memorising letter names of high-frequency sight words. It is targeted to the 
needs of individual learners, identified through a bank of assessments for learning, and involves two 
individual 15-minute sessions per week. A member of staff within each school manages Catch Up 
Literacy while the intervention is usually delivered by trained teaching assistants. Catch Up provides a 
four-part training programme for managers and classroom assistants, alongside ongoing support from 
the Catch Up trainer. It is intended that managers attend all four parts of the training, while classroom 
assistants attend two. The training programme includes: 

• LI ‘Introducing Catch Up Literacy', a 90-minute session for senior managers and the member 
of staff managing Catch Up. 

• L2 ‘Delivering Catch Up Literacy’, three half-day sessions for staff who will deliver the 
intervention and the member of staff who will manage it. 

• L3 ‘Managing Catch Up Literacy’, a 60-minute session for the member of staff who will be 
managing the intervention. 

• L4 ‘Review and next steps’, a half-day session for those who are delivering the intervention 
and the member of staff managing it. 

There are four stages of Catch Up Literacy: assessments for learning, which are used to set targets 
and identify the appropriate starting points for pupils; selecting an appropriate book for the learner to 
read; delivery of two individual (one-to-one) sessions per week, each lasting 15 minutes; and ongoing 
monitoring, through which assessments for learning are revisited and targets are reviewed. In this 
evaluation, the Catch Up Literacy intervention lasted 30 weeks and sessions followed the following 
schedule: 

• three minutes of prepared reading 

• six minutes of the pupil reading out loud 

• the pupil and TA then discuss the text 

• six minutes of linked writing. 

Catch Up Literacy was launched in 1998, based on original research by Diana Bentley, Suzi Clipson- 
Boyles and Dee Reid. It was designed for seven- to eight-year-old (Year 3) pupils who only achieved 
level 1 for reading in the Key Stage 1 Standard Assessment Tests, but has since been developed for 
use in secondary schools (Catch Up, 2008). It is now appropriate for learners from 6 to 14 years of 
age who struggle with reading. 

Background evidence 

The approach has been informed by a range of research evidence relating to literacy. This includes 
evidence related to supporting struggling readers (Bentley and Reid, 1995), the influence of children’s 
attitudes to reading on their progress in learning to read (Wray and Medwell, 1991), matching 
struggling readers to books that are sufficiently challenging but not frustrating (Kress and Johnson, 
1965), the importance of fluency and reading for meaning (Stanovich,1980), the reciprocal gains of 
reading and spelling (Clay, 1991), and taking a known sight word as a starting point for analogies in 
reading development (Goswami, 1994). 

The Catch Up Literacy intervention was designed with these findings in mind. For example, stage 1 of 
the intervention (the assessment) includes a reading interview which assesses pupils’ attitudes 
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towards, perceptions of, and experiences of reading. During the sessions, the prepared reading 
element is designed so that the pupil is able to concentrate on reading for meaning, while the linked 
writing element, which follows the reading elements, allows the pupil to benefit from the reciprocal 
gains of reading and spelling. 

Since its development there have been a number of studies reporting on the benefits of Catch Up 
Literacy. For example, a study looking at Catch Up Literacy in 28 secondary schools in England and 
Wales (427 pupils in Years 7 to 10) reported that, after an average of 8.3 months on Catch Up, pupils’ 
reading age increase by an average of 16.71 months (Catch Up, 2008). However most of these 
studies did not include a control group and so their results should be treated with caution. 

In 2012, Holmes et al. undertook a small-scale trial of Catch Up Literacy with 87 pupils in Years 7 and 
8 in one local authority in England. A treatment group of 20 pupils were given Catch Up Literacy 
support while a control group of 67 pupils were given matched-time support (in which they received 
additional literacy support for the equivalent time as the Catch Up group but not Catch Up Literacy). 
Pupils were randomly assigned to either of the two groups. Pupils’ reading level was measured before 
starting the trial, which showed there were no significant differences between the two groups, and 
again after the intervention was completed. The results showed that learners receiving Catch Up 
support made a mean gain of 13.10 months in terms of chronological reading age (ratio gain 2 of 3.27), 
compared to 5.57 months in the time-equivalent group (ratio gain of 1.39). The effect size (based on 
the difference in the ratio gains of the two groups divided by the pooled standard deviation) was seen 
to be large (d=0.86). However, these results are based on very small numbers and were not 
statistically significant. 

Overall, the intervention is based on an evidence-based approach to literacy and a small-scale trial 
has shown promising results. Therefore, the research behind the principles used by Catch Up Literacy 
is fairly strong, but there is not yet compelling RCT-based research evidence that the programme 
itself has had a significant effect on raising literacy levels. 

Evaluation objectives 

The primary research question was to identify the impact of the Catch Up Literacy intervention on the 
reading ability of individual pupils over a 30-week intervention period. The intervention period included 
the summer holiday. Pupils selected for the intervention started receiving the intervention at the end 
of Year 6 and started again when they joined Year 7. Pupils' literacy skills were measured by 
performance on the New Group Reading Test. 

The process evaluation explored the implementation and scalability of the intervention. 

Project team 

The intervention delivery evaluation team was led by Dr Ann Dowker of the University of Oxford and 
Dr Graham Sigley from Catch Up. This team was responsible for the recruitment of schools and TAs, 
the training of TAs to use the Catch Up intervention, liaising with schools to organise, providing test 
administration guidance, and undertaking follow-up sessions with TAs at the end of the intervention. 

The external evaluation team at NFER was led by Simon Rutt, Head of Statistics. While the overall 
project and impact evaluation were led by Simon Rutt, the process evaluation was led by a number of 
researchers over the lifetime of the project starting with Claire Easton and Claire O’Beirne and ending 
with Kelly Kettlewell and Gill Featherstone. They were all either Researchers or Senior Research 
Managers within the Research Department. Within the impact evaluation the NFER team was 

2 Ratio gain is the gain in reading age made during a chronological time span, expressed as a ratio of that time 
span. A ratio gain of one would mean that the pupil's skills are developing at normal pace. 
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responsible for the randomisation of pupils, the analysis of test data and the writing of a final 
evaluation report. NFER was responsible for the whole of the process evaluation and its contribution 
to the final report. 

Ethical review 

The NFER has a well-developed Code of Practice that contains detailed ethical protocols. These 
protocols govern all research undertaken by NFER and this trial lies within them. Parents gave active 
written consent for all eligible pupils put forward for the intervention and testing, and the Catch Up 
team confirmed that consent had been received before continuation within the trial. 
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Methodology 

Design 

This was a multicentre, two-arm, parallel-group, pupil-randomised trial within schools in England. The 
trial was designed to involve 17 secondary schools with two part-time teaching assistants (TAs) being 
identified from within each secondary school. The EEF grant covered salaries and other costs to 
schools. These TAs worked with all feeder primary schools that contained treatment pupils. As the 
Catch Up intervention is delivered on a one-to-one basis there was a very small risk of cross- 
contamination between treatment and control groups, meaning that pupil randomisation is the 
preferred design as it is more efficient than a clustered design. 

Apart from the number of secondary schools recruited to the project, which was reduced from 17 to 
15, the design of the trial remained consistent with the design outlined in the published protocol. 3 

Eligibility 

Pupils were selected as eligible by their feeder primary schools if their predicted level in the Key 
Stage 2 Reading assessment was below a level 4b. Lists of pupils were provided to NFER for the 
purpose of randomisation and the evaluator was not aware of the methods used by schools to select 
pupils for these lists. NFER does not believe this was carried out in a way that would introduce bias 
into the selection of pupils. Lists of pupils provided to NFER were first checked for correct eligibility 
criteria, i.e. if any pupils had a predicted level above 4c they were removed from the list. Parental 
consent had to be obtained by the school and confirmation that this had been received was indicated 
on the file provided to NFER. NFER was not responsible for obtaining this consent but has no reason 
to believe that this was not undertaken in an acceptable manner and was received for all pupils prior 
to randomisation. A copy of the consent letter is contained in Appendix D to this report; while NFER 
has no direct evidence that these were received for all pupils, schools were instructed to only provide 
pupil details if this consent had been received. 

Eligibility was also dependent on the pupil having obtained a place at the relevant secondary school. 
This would have been known at the time of selection although a pupil may still not actually transfer to 
that secondary school. Indeed, 25 pupils were excluded from the analysis as they did not transfer to 
relevant secondary schools or left their school before the end of the 30-week period. These pupils are 
highlighted later in the report. 

Intervention 

Catch Up Literacy is a structured one-to-one literacy intervention for learners aged 6 to 14 who are 
struggling to learn to read. The intervention adopts a combination of segmenting and blending 
phonemes, and memorising letter names of high-frequency sight words. It is targeted to the needs of 
individual learners, identified through a bank of assessments for learning, and involves two individual 
15-minute sessions per week. How the sessions were run in practice is discussed at greater length in 
the process evaluation section of this report. Pupils were identified while in primary school and feeder 
secondary schools employed two part-time teaching assistants (TAs) to deliver the intervention. Two 
assistants were recruited from within each secondary school to ensure the project was able to deliver 
the correct number of sessions to those pupils selected to be part of the intervention. Teaching 
assistants organised the delivery of the sessions directly with the primary schools, ensuring any 
burden on the schools was kept to a minimum. 


3 The Catch Up Literacy evaluation protocol is located here: 

http://educationendowmentfoundation.org.uk/uploads/pdf/Transitions-_Catch_Up.pdf 
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A number of sessions were held within the primary schools and pupils were re-identified when they 
transferred to their feeder secondary school. Due to slightly different start times, TAs worked within 
primary schools for between four and five weeks. A number of pupils did not transfer as expected and 
they are identified in the flow diagram. 

The intervention itself is made up of four main stages. 

Stage 1 - Assessments for learning 

• A bank of assessments that can be used to determine what the learner can do and where 
their needs lie. 

• These assessments are used to set literacy targets and the appropriate starting point. 

Stage 2 - Selecting an appropriate book 

• Catch Up has an online list of more than 8,000 books that have been categorised into 12 
gradually increasing levels of difficulty. 

• Books are selected with the aim of enabling a struggling individual to read with a high degree 
of confidence and success. 

Stage 3 - Individual 15 minute sessions 

Individual session part 1: Prepared reading (3 minutes) 

This part of the session aims to: 

• give the learner an overview of the story, so that they can concentrate on reading for meaning 

• introduce unfamiliar vocabulary 

• give the learner more confidence to tackle the text. 

Individual session part 2: The learner reads and text is discussed (6 minutes) 

This part of the session aims to: 

• give an opportunity to identify which reading strategies the learner uses 

• encourage the learner to take responsibility for tackling any less familiar words 

• provide an opportunity to discuss the text and to ensure that the learner understands the 
content and can infer meaning and express opinions. 

Individual session part 3: Linked writing (6 minutes) 

This part of the session aims to: 

. provide focused support based on observed miscues 

• enable the learner to benefit from the reciprocal gains of reading and spelling. 

Stage 4 - Ongoing monitoring 

• A record of each session provides information about the learner’s needs and progress. 


Education Endowment Foundation 


10 



Catch Up Literacy 


Outcomes 

The primary outcome was reading ability, and was estimated using the New Group Reading Test 4 
(NGRT). This assessment was chosen by the EEF, in consultation with NFER, as it is a reliable and 
accepted assessment of literacy and was the test used across all 23 trials in the EEF's literacy catch- 
up funding round. Secondary outcome data was collected by NFER through a pupil questionnaire 
administered during the same session as the final NGRT assessment (see appendix A for the 
questionnaire). The questionnaire was designed by NFER researchers using items that had shown 
reliability in previous studies. NFER collated all responses and created five composites that identified 
pupils’ perception in the following areas: 5 

Attitude to school 

1. It is important to do well at school 

2. I try hard at school 

3. Most of the time I feel confident about doing my work in lessons 

4. Most of the time I enjoy school 

5. I behave well in lessons 

6. I always do my homework 

Self-esteem 


1 . I feel happy 

2. I am a confident person 

3. I have people to talk to if I feel unhappy or worried 

4. I like the way I am 

5. I feel positive most of the time 

Like reading 


1 . I enjoy reading 

2. I enjoy reading in my own time 

3. Reading is one of my favourite things to do 

4. I enjoy talking to my friends/family about the books I have read 

Good at reading 

1 . I feel confident reading out loud to the class 

2. I know how to deal with words I find hard to read 

3. I am doing well in reading 


Good at writing 


1 . I enjoy writing 

2. I find writing easy 

3. I feel confident about writing in sentences 

4. Writing is one of my favourite things to do 

5. I am doing well in writing 


4 The New Group Reading Test (NGRT) is used in groups to assess and monitor reading and comprehension. More information 
can be found at http://www.gl-education.com/international-products/new-group-reading-test-ngrt 


5 Reliability analysis was carried out on all composites and the following Cronbach alpha's were achieved: attitude to school 
0.8, self-esteem 0.8, like reading 0.9, good at reading 0.6, and good at writing 0.8. 
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These secondary outcomes were deemed appropriate by EEF as Catch Up Literacy could produce 
desirable outcomes in addition to the reading ability measured by the NGRT. 

Tests and questionnaires were administered by the teaching assistants and all schools were provided 
with the teacher guidance supplied by the NGRTs and additional guidelines from Catch Up that were 
relevant to this trial. These guidelines were seen by the external evaluator prior to them being sent to 
the schools and covered the administration of the tests, their timings, whether breaks were allowed, 
how they should be managed, testing of absent pupils, and how to return completed papers. 

All test papers were sent to Catch Up and then to Dr Dowker who marked all papers and generated 
raw and standardised scores. Dr Dowker did not know the membership of each group until she had 
finished marking the post-test data, as group membership was added by the Catch Up team after 
marking. All data was sent to NFER via a secure data transfer portal. 

The independent evaluator checked the guidelines for test administration, but otherwise the process 
of test administration and test marking was not evaluated by the evaluator. There is no evidence to 
suggest that testing was not conducted according to appropriate protocols. 

Sample size 

The sample size was determined by the intervention provider prior to the appointment of NFER to the 
role of external evaluator. The original design was for 17 schools to provide 408 pupils in both 
treatment and control groups, 48 pupils per school. Based on the assumptions of a minimal intra class 
correlation of 0.05, the original design has a Minimum Detectable Effect Size (MDES) of 0.18. 

The final design was with 15 schools containing 286 pupils in the treatment group and 271 in the 
control group. Power calculations on the basis of the final sample were carried out. In calculating the 
power of the design, the actual intra-class correlation (rho) was 0.06. Given the sample size and 
assumptions on rho, this design has a power of <0.80 and is therefore underpowered to detect an 
effect size of 0.12. The MDES for this design is 0.27. 



Effect size 
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Randomisation 

An Excel-based randomisation algorithm was used by NFER to allocate eligible pupils to one of three 
groups. These were the treatment and control groups and a reserve group if a secondary school was 
able to provide more than the 48 pupils requested. Given the number of pupils who needed to receive 
the treatment it was decided by Catch Up that 48 pupils was the maximum that one secondary school 
could provide for the trial. At the start of the intervention pupils were registered in a number of 
different feeder primary schools. These pupils would be randomly allocated to one of the two groups. 
This would maintain a balanced design with a maximum of 24 pupils from one school in each group. 

The pre-test measure was to be the pupils’ end of Key Stage 2 National Curriculum reading results; 
since the randomisation procedure took place before the pupils sat their end of KS2 tests, there was 
potential for this group to be reduced in size due to absence from the KS2 testing process. To 
maximise the number of pupils, it was decided to ask secondary schools to provide up to 60 eligible 
pupils with 48 being randomised to treatment or control and up to 12 being randomly allocated to a 
reserve group. If a pupil who had been selected for the treatment or control group was then absent on 
the day of their KS2 assessment they would be replaced with a pupil from the reserve list. Of the 21 
pupils who had been allocated to the reserve group, only three of these pupils entered the project 
before the trial commenced. Instructions were sent by Catch Up to primary schools on how their 
pupils were selected, and into which group they had been selected, but there was not a random 
selection from within the reserve group if it was necessary to replace a pupil who had not sat their 
KS2 assessment. 

Very few schools were able to provide more than 48 eligible pupils, and where a school provided 
fewer than 48 pupils, all pupils were allocated to the treatment or control groups. While every attempt 
was made to maintain a balanced design with the same number of pupils in treatment and control, in 
some schools this was not possible as consideration had to be given to the contracts given to the 
TAs. Where schools provided less than 48 eligible pupils, the randomisation process allocated a 
number to be randomised to the intervention group with the remainder to the control group. For 
example, a school that provided 33 pupils may have had 19 randomised to the treatment group and 
14 to the control group. Pupil lists were provided to NFER by Catch Up and following randomisation 
lists were returned to Catch Up so that they were able to inform the TAs of the location of all treatment 
pupils. This was necessary to minimise the travelling time between primary schools and to maximise 
the amount of time primary schools had to organise their class teachers. Following KS2 assessments, 
TAs were informed of the pupil names so that they could develop more detailed plans with primary 
schools. 

Analysis 

The primary aim of the analysis is to assess whether the Catch Up Literacy intervention had a 
significant impact on pupils’ literacy, as measured by the post-intervention test scores and controlling 
for prior attainment in the form of Key Stage 2 reading test scores. This is performed on an intention- 
to-treat basis. That is, the original random assignment to treatment and control groups is reflected in 
the analysis, regardless of whether the pupil actually received the intended intervention. 

The primary analysis examined the results of the post-intervention test, controlling for prior attainment 
and treatment assignment. Additionally, the analysis controlled for age, gender, and free school meal 
(FSM) eligibility, together with interaction terms for gender with treatment, and for FSM eligibility with 
treatment, to assess the extent to which the intervention has had a differential impact across 
subgroups. This analysis was performed using a simple linear regression model, with school fixed 
effects controlled for by including individual school dummy variables as covariates. 6 While the protocol 

6 Considering the small number of schools involved, and the fact that pupils were randomised within schools, controlling for 
school effects with the use of school dummies is preferable to using multilevel models in this context. 
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identified multi-level modelling would be used for analysis, the above analysis of dummy variables to 
identify schools was used while the analytical program STATA adjusted the standard errors to take 
account of the clustered nature of the data. As required by EEF policy, the same analysis was also 
performed separately for the group of FSM pupils. 

The secondary analysis looked at the outcomes from the pupils' questionnaire (see Appendix A). The 
items in sections 2 and 3 were specifically designed to derive composite measures on, respectively, 
general attitudes towards school and levels of confidence/self-esteem of the pupils. Additionally, we 
performed factor analysis on the items in question 1 to identify one or more composite measures 
regarding specific attitudes towards reading and writing. The structure of the analysis of composite 
outcomes was similar to that of the primary analysis, with a simple specification including prior 
attainment and treatment assignment, and a further specification including age, gender, and FSM 
eligibility as additional covariates, together with interaction terms for gender with treatment, and for 
FSM eligibility with treatment. As with the primary analysis, the analysis was performed using a simple 
linear regression model, with school fixed effects controlled for by including individual school dummy 
variables as covariates. 

Effect sizes are calculated using Hedges’ g formula as described in the EEF analysis guidelines. All 
the preparatory data processing is performed in IBM SPSS 21, while the analysis is performed using 
STATA 13. 

Process evaluation methodology 

The process evaluation comprised three different components: observations of training sessions; 
qualitative interviews with coordinators and TAs; and surveys of TAs at two time points. The first 
component consisted of the team attending and observing Catch Up Literacy training sessions. The 
team attended ‘Delivering Catch Up Literacy’ (L2) and ‘Managing Catch Up Literacy’ (L3) training 
sessions prior to the intervention commencing in May 2013. The team then observed two follow-up 
‘Review and next steps’ (L4) sessions during February and April 2014. 

For the second component, the team undertook telephone interviews with 13 TAs, 7 Catch Up 
Literacy coordinators, and 2 TAs who were both coordinating and delivering Catch Up Literacy (N= 22 
participants, across 13 participating schools). Each school has a Catch Up Literacy coordinator who 
oversees the intervention within the school and offers support to the TAs. Researchers chose this 
data collection method in order to gain in-depth qualitative data from a wide range of interviewees 
during a short timescale while ensuring disruption to the participating schools, TAs, and coordinators 
was kept to a minimum. Interviews took place between June and July 2014. 

The team invited all TAs and coordinators to participate in a semi-structured interview with the aim of 
achieving up to 25 participants in total. However, the timing of the interviews meant that recruiting TAs 
and coordinators was challenging. Furthermore, their availability was limited due to end of school-year 
activities. 

Three experienced NFER researchers carried out all interviews via telephone. Interviews took 
between 15 to 35 minutes, with most taking around 20 minutes. As per NFER’s Code of Practice, at 
the start of every interview the researchers explained the purpose of the interview, and the data 
protection and confidentiality protocols. Interviews were recorded only when interviewee permission 
was granted. 

For the final component of the process evaluation, Catch Up teaching assistants were sent paper- 
based surveys at two time-points (midway through the intervention (Appendix B) and at the end of the 
intervention (Appendix C)). At both time points the surveys were sent to all 28 TAs delivering Catch 
Up. The aims of the survey were to assess how the training prepared the TAs to deliver the 
intervention and to ascertain how the intervention had actually been delivered and whether the fidelity 
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of the trial had been maintained. In total we received 22 completed surveys at the mid-point of the 
intervention and 27 completed surveys at the end of the intervention. These are response rates of 
62% and 96%. The response rate was lower than hoped for in the first survey, but was very high in 
the second survey. 
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Impact evaluation 

Timeline 


Activity 

Date 

Schools recruited 

January-March 2013 

Schools provide list of eligible pupils to NFER 

By end of March 2013 

Schools informed of randomised groups 

April 2013 

Pupils sat KS2 assessments 

May 2013 

Trial commences 

June 2013 

Pupils transfer to secondary school and sit NGRT test 

September 2013 

Pupils complete trial 

March 2014 

Pupils sit NGRT post test 

March-April 2014 

Data provided to NFER for analysis 

July 2014 


Schools 

The external evaluator was only appointed after this recruitment had taken place so had no input into 
the process. There is no reason to believe that it was not undertaken in an appropriate manner. The 
following is the process adopted by Catch Up in obtaining secondary schools for the project. NFER 
are unaware of the total number of schools approached. Catch Up adopted the following protocols 
when recruiting schools. 

• Any secondary school that contacted Catch Up between January and June 2013 enquiring 
about either of the Catch Up interventions was offered the opportunity to be part of the project 
if they could get their feeder primary schools involved. 

• Catch Up approached its local authority contacts and asked if they had schools in their local 
authority area that were in challenging contexts and could benefit from intervention support. 

• Catch Up directly approached some schools. 

There were 15 secondary schools involved in the trial, 7 maintained schools and 8 academies. 7 The 
majority of schools were located in urban areas, with 4 located in smaller towns and 1 in a rural area. 
The three main regions of England were all represented, with 9 schools in the South (including 1 in 
London), 2 schools from the Midlands, and 4 from the North. 

As shown in the tables below, the majority of schools were rated Good or Outstanding by Ofsted, and 
they were reasonably well distributed across quintiles of FSM rates. 


7 As of start of academic year 2013/14. 
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Table 1 : Number and percentage of participating schools by Ofsted rating 



Frequency 

Percent 

Outstanding 

3 

20.0% 

Good 

8 

53.3% 

Requires Improvement 

1 

6.7% 

Inadequate 

3 

20.0% 

TOTAL 

15 



Table 2: Number and percentage of participating schools by FSM quintiles 



Frequency 

Percent 

Lowest 20% 

3 

20.0% 

2 na lowest 20% 

4 

26.7% 

Middle 20% 

3 

20.0% 

2 na highest 20% 

3 

20.0% 

Highest 20% 

2 

13.3% 

TOTAL 

15 



Pupils 


As shown by the flow diagram below, 631 pupils were randomised into three groups: the Catch Up 
(treatment) group (315 pupils), the control group (295 pupils) and the reserve group (21 pupils). Of the 
21 reserves, 3 were eventually included in the treatment group. 

The final analysis comprises 286 pupils for the treatment group and 271 pupils for the control group. 
Of the 32 excluded from the treatment group, 5 were missing end of Key Stage 2 test results and 27 
had no post-intervention test. The reasons for missing the post-intervention test are as follows: 17 
pupils did not transfer to, or eventually left, the participant school; 6 were withdrawn from the trial; 1 
pupil was absent; and 3 had other reasons. Of the 24 pupils excluded from the control group, 8 were 
missing Key Stage 2 results and 16 had no post-intervention test. The reasons for missing the post- 
intervention test are as follows: 8 pupils did not transfer to, or eventually left, the participant school; 2 
pupils were absent; 1 was withdrawn from the trial; and 5 had other reasons. Finally, 18 of the original 
21 pupils in the reserve group were never included in the trial. These pupils commenced the trial at 
the beginning with all other pupils. 

An analysis of the characteristics of the 56 pupils excluded from the analysis shows that there is no 
significant difference in the gender composition compared to the group of pupils included in the 
analysis, whereas the percentage of pupils eligible for free school meals is significantly higher (40%, 
compared to 21% for the pupils in the analysis). The difference in KS2 reading points scores between 
the two groups is also not statistically significant, with an effect size of 0.09 (C.l. -0.21 - 0.40). 
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Lost to follow-up: 


Lost to follow-up: 

Original allocation (n=27) 


Follow-Up 


Original allocation (n=16) 

Added reserves (n=0) 




Added reserves (n=0) 


T 

Analysed (n=286) 

Excluded (no prior attainment) (n=5) 

Total excluded (including lost to follow-up): 
(n=32) 
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Pupil characteristics 

Tables 3, 4, and 5 compare pupil characteristics for the treatment and control groups, and show they 
are broadly similar in terms of gender composition, percentage of pupils eligible for FSM, and prior 
attainment. Statistical tests 8 show no significant differences between the two groups, at the standard 
95% level of confidence, for the characteristics in question. The baseline effect size for Key Stage 2 
reading score is not statistically different from zero, with 0.11 (C.l. -0.06; 0.27) standard deviations in 
favour of the control group. The baseline effect size for the NGRT test score taken at the start of Year 
7 is also not statistically different from zero, with 0.02 (C.l. -0.19; 0.14) standard deviations in favour 
of the treatment group. 


Table 3: Gender by randomised group 



% Male 

% Female 

N 

Original Randomisation 

Control 

59.3% 

40.7% 

295 

Treatment 

61.3% 

38.7% 

318 

All pupils 

60.4% 

39.6% 

613 

Main Analysis 

Control 

57.9% 

42.1% 

271 

T reatment 

60.8% 

39.2% 

286 

All pupils 

59.4% 

40.6% 

557 


Table 4: Eligibility for free school meals by randomised group 



% No 

% Yes 

N 

Original Randomisation* 

Control 

75.3% 

24.7% 

288 

T reatment 

80.3% 

19.7% 

314 

All pupils 

77.9% 

22.1% 

602 

Main Analysis 

Control 

77.1% 

22.9% 

271 

T reatment 

81.5% 

18.5% 

286 

Total 

79.4% 

20.6% 

557 


*not all pupils had FSM information 


8 The statistical tests used in this instance were a test of proportions for the categorical variables gender and FSM eligibility, 
and a t-test for the prior attainment and test variables. 
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Table 5: Average prior attainment by randomised group 



KS2 Reading 
Points Score 

NGRT Test Score 
(Sept 13) 

N 

Original Randomisation 

Control 

24.8 

NA 

287 

T reatment 

24.2 

NA 

313 

All pupils 

24.5 

NA 

600 

Main Analysis 

Control 

24.7 

89.3 

271 

T reatment 

24.2 

89.6 

286 

All pupils 

24.4 

89.5 

557 
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Outcomes and analysis 

Primary analysis 

The primary aim of the analysis is to assess whether the intervention has had any significant impact 
on pupils’ literacy, as measured by the post-intervention test scores, controlling for prior attainment in 
the form of Key Stage 2 reading scores. Tables 6 and 7 show the results of the main analysis, 
conducted on an intention-to-treat basis. 


Table 6: Intention-to-treat analysis outcomes (basic specification) 



Model 

coefficient 

Standard 

error 

95% Cl 

P 

Catch Up Treatment 

1.24 

0.72 

-0.17-2.65 

0.09 

Intercept 

56.34 

2.96 

50.53-62.15 

0.00 

KS2 Reading Score 

1.28 

0.08 

1.13-1.43 

0.00 


Table 6 shows the results of the basic specification, with prior attainment and the intention to treat 
(ITT) indicator as the main explanatory variables, together with school-level dummies to control for 
clustering of pupil results. Estimates for the main covariates are reported in the table. 9 

The results show that the post-test score for pupils in the treatment group is, on average, 1.24 points 
higher than the score of pupils in the control group. However, this difference is not statistically 
significant at the 95% confidence level. The estimated effect size is 0.12 (C.l. -0.02 - 0.25). 

The results of the analysis including additional pupil characteristics are reported in Table 7. The 
additional explanatory variables are gender, age (in months), and FSM eligibility, together with 
interaction terms for gender and treatment, and for FSM eligibility and treatment. These results, while 
broadly confirming those of Table 6, do not identify any significant interaction. The effect size for this 
analysis is 0.16 (C.l. -0.02 - 0.34) 

As required by all EEF evaluations, we performed a separate analysis for FSM pupils, with the same 
specification as above. This subgroup analysis does not identify any significant effect for the 
explanatory variables included, except for the prior attainment measure. The estimated effect of the 
treatment among FSM pupils is negligible, with an effect size of 0.00 (C.l. -0.30 - 0.30). The cohort 
sizes for this analysis were 62 FSM pupils in the control group and 53 pupils in the treatment group. 


9 The sign and size of the individual school-level dummies vary across schools, but the estimates are never 
significant at the 95% confidence level. 
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Table 7: Intention to treat analysis outcomes (subgroup specification) 


Model 

coefficient 

Standard 

error 

95% Cl 

P 

Catch Up Treatment 

1.71 

0.99 

-0.23-3.66 

0.08 

Intercept 

111.38 

15.71 

80.52-142.24 

0.00 

KS2 Reading Score 

1.30 

0.08 

1.15-1.45 

0.00 

Gender (Female = 1, Male = 0) 

0.68 

1.04 

-1.37-2.72 

0.52 

Age (in months) 

-0.39 

0.11 

-0.60- -0.18 

0.00 

FSM (eligible = 1, Not eligible = 

0.76 

1.29 

-1.77-3.29 

0.56 

0) 





Female in treatment group 

-0.64 

1.46 

-3.50-2.22 

0.66 

FSM in treatment group 

-1.38 

1.77 

-4.85-2.09 

0.44 


An on-treatment analysis was undertaken using sessions data provided by Catch Up. Data was 
available for the majority of pupils with a mean number of sessions being 32, ranging between 6 and 
52. Sessions data was introduced into the models instead of the dichotomous variable that indicated 
pupil membership of either treatment or control groups. Pupils in the control group were given a 
dosage of zero. No significant effect was found when using this session’s data. At the request of 
Catch Up a dichotomous variable was also introduced which identified whether a pupil received less 
than 30 sessions or more than 30 sessions. This variable also was non-significant. 

Some additional analysis was requested by Catch Up and was therefore undertaken outside the 
original protocol. Analysis was undertaken to assess the effect of prior experience of the school with 
the Catch Up programme on the post-intervention test scores. While the results show a positive effect 
size of 0.48 (C.l. -0.02 - 0.98), this is not statistically significant, due to the high variability of results. 
Adding an interaction term indicating pupils in the treatment group within these schools yields a 
negligible effect. 

Analysis also looked at the differential effect for pupils above and below level 4B at Key Stage 2. 
Adding a further interaction term to the model reported in Table 7 shows that pupils with level 4B or 
above are substantially more likely to have benefited from the intervention, with an effect size of the 
interaction term of 0.4 (C.l. 0.19 - 0.61). Performing the analysis separately for pupils with level 4C or 
below, using the primary analysis model specifications, we find an effect size of 0.13 (C.l. -0.05 - 
0.32), similar to that of the main analysis but not statistically significant, due to the high variability of 
results. None of the interaction terms are found to be significant. 

Secondary analysis 

The composite measures described in the methodology section are used as outcome variables for the 
analysis presented below. Unfortunately, we were not able to match questionnaire data with results 
data for 56 of the 557 pupils included in the primary analysis. However, this does not appear to 
generate a systematic bias in the analysis. 10 

Table 8 below shows the estimated effect size of the Catch Up intervention on the five composite 
measures, together with its 95% confidence interval and p-value. 

The results show a statistically significant effect on a number of these outcomes with the largest effect 
being on reading confidence and ability, with an effect size of 0.32 (C.l. 0.15 - 0.50). Also significant 
were the effects on writing confidence and enjoyment, and general attitude towards school. 

10 There is no systematic difference in gender composition, FSM rates, or in prior attainment, as measured either by KS2 
results or by pre-intervention test results. 
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Interaction terms were only significant for the measure identifying attitude towards school, with 
females and FSM pupils showing a smaller, although still positive, effect of the intervention. The 
intervention effect was 0.51 while the FSM/Intervention interaction had a negative effect of -0.45. 


Table 8: Secondary analysis: effect of intervention on questionnaire composites 


Composite 

Effect size 

95% Cl 

P 

Positive attitude towards school 

0.25 

0.08-0.42 

0.00 

Confidence and self esteem 

0.14 

-0.03-0.32 

0.10 

Reading enjoyment 

0.14 

-0.03-0.31 

0.11 

Reading confidence and ability 

0.32 

0.15-0.50 

0.00 

Writing confidence and 
enjoyment 

0.23 

0.06-0.40 

0.01 


Cost 

All of the direct costs to schools were borne by the EEF grant for the project. The evaluator used this 
information and the number of pupils receiving the intervention to estimate that Catch Up Literacy 
cost an average of £769 per pupil in this evaluation. This estimate includes the salary costs of TAs 
and the cost of the training provided by Catch Up. This cost will be lower if more pupils are involved in 
the intervention, as the cost of training a TA will be spread out over more pupils. The cost will also be 
lower if schools do not implement Catch Up Literacy as an additional activity that requires paying TAs 
for more time, but use it to replace things they were already doing, as this will eliminate the salary 
cost from the cost estimate above. The training provided by Catch Up costs £350 per attendee. The 
independent evaluator did not look at additional costs that were associated with attendance at this 
training or additional resources that may have been required to deliver the intervention successfully. 

In 2010, the Catch Up Literacy developers commissioned New Philanthropy Capital (NPC) to 
undertake a cost evaluation, which estimated that the cost per struggling learner is £120. This was 
later updated to £130 to account for inflation. Further details can be found at 
http://www.catchup.org/Training/CatchUpLiteracytraining.aspx. 
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Process evaluation 

Implementation 

All TAs interviewed had attended the three half-day L2 training sessions which focused on delivering 
Catch Up Literacy. However, two of the coordinators had not attended the training; one did not attend 
because they had received the training previously before they had started to work at the school, while 
another was told by their school that they were not allowed to attend. Nearly all interviewees found 
the L2 training very useful, thorough, and delivered at an appropriate pace. Three TAs, who were 
already delivering Catch Up, felt that the training sessions helped to refine what they were already 
delivering. Most coordinators believed that it was cost effective for the coordinators to attend the L2 
training sessions as it was important for them to be able to understand the delivery in order to support 
and oversee the work of the TAs. 

Nearly all of the coordinators attended the L4 training session at the end of the intervention, while all 
TAs had attended this session. This was also seen as a positive training session as it helped the TAs 
to share knowledge and offered reassurance of their delivery. Two coordinators and one TA felt that 
this session was helpful but not essential for delivery. 

When asked if they had spent additional time familiarising themselves with the materials prior to 
delivery, most of the TAs said yes. This varied from spending very little time re-reading sections of the 
booklet to spending several hours in their own time familiarising themselves with the materials, getting 
used to the different levels, and learning how to set targets. Coordinators generally had not spent 
additional time familiarising themselves with the materials. 

Both the TAs and coordinators valued the support that Catch Up had provided to them, in addition to 
the training sessions. All TAs felt the trainers were very supportive and approachable outside of the 
sessions and many of the TAs explained that they often contacted their trainer with questions as they 
started with delivery. The trainers' visits to the schools were seen as particularly valuable. 

Generally TAs delivered up to three Catch Up sessions per hour to pupils. However, they believed 
that delivering two Catch Up sessions per hour was more realistic. This was because TAs needed to 
prepare prior to the session and complete paperwork after the session. In some instances, the TAs 
also needed the flexibility to be able to go and collect pupils who had forgotten to attend the session. 
A small number of TAs explained that they had planned to deliver either four or three sessions in an 
hour, which was later changed due to the timing being too tight. In line with this, when asked about 
time spent preparing for sessions, 17 of the 27 TAs (52%) who completed the end of intervention 
survey stated that they spent at least 10 minutes preparing for each of the sessions. 

Overall, TAs felt well equipped to deliver Catch Up Literacy, and this seemed to increase with time. At 
the mid-point of the intervention, 16 of the 22 TAs (73%) said that they felt equipped to deliver Catch 
Up Literacy either to a great extent or entirely, while 6 said to some extent, and none said not at all. 
By the end of the intervention all of the 27 TAs who responded to the questionnaire said that they felt 
equipped to deliver Catch Up Literacy either to a great extent or entirely. It should be noted that two 
TAs and six schools had previous knowledge of Catch Up. 

Both TAs and Catch Up coordinators who were interviewed identified a number of barriers to 
delivering Catch Up Literacy. The most common barrier was a lack of time, specifically relating to TAs 
not having sufficient time to prepare and deliver the intervention, and timetabling restrictions. Some 
interviewees also said that sometimes pupils did not turn up for sessions, which made it more difficult 
to fit in all sessions as these then needed to be rescheduled. 

TAs also highlighted the difficulty in finding a suitable location within the school to deliver the 
intervention which was sufficiently private so that pupils did not get distracted or feel embarrassed by 
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being seen getting extra support, particularly where they were expected to read out loud. Two 
coordinators said that, due to previous experience of running literacy interventions (including Catch 
Up), they felt it was very important to have a designated private room for the intervention, and as such 
had ensured their TAs had this resource. 

Two TAs and two coordinators said that they had come across some resistance from staff members 
who did not want the pupils to be taken out of their lessons. This made timetabling more difficult. One 
TA, who was also a coordinator, said that they did not have the buy-in from the senior management 
team, as they preferred interventions that could be run as a group rather than a one-to-one 
intervention. 

Other barriers mentioned by a minority of interviewees included: 

• primary schools not wanting to engage with the intervention 

• issues relating to the funding of the intervention such as not receiving enough funding for the 
number of staff involved or not receiving the funding before starting delivery 

• keeping the intervention as a transition project after the end of the trial, due to funding and 
logistical constraints (for example a lack of funding for TA travel expenses). 

Fidelity 

Most of the schools delivered Catch Up Literacy sessions in timetabled slots whereby pupils were 
taken out of the same lesson each week (or every other week if the school operated a two-week 
timetable). A small number of schools tried to avoid pupils missing the same lessons by changing the 
times of the Catch Up sessions each week. In three cases, the schools tried to avoid pupils missing 
any lessons by running Catch Up sessions before school or in additional literacy time or tutor time. 

When planning delivery, most schools tried to avoid timetabling sessions to coincide with certain 
lessons, including core lessons such as mathematics, English or science, or, for practical reasons, 
physical education and drama. In the majority of schools interviewed, the school did not want the 
pupils to miss core curriculum; however, in a small number of cases the school took into account the 
subjects pupils enjoyed as they were concerned that the pupils would be less likely to engage with the 
Catch Up sessions if it meant they would miss their favourite lessons. 

The majority of TAs interviewed said that they were able to deliver two Catch Up sessions per week. 
In one case the school had a two-week timetable and this caused timetabling issues which resulted in 
some of the pupils having three sessions in the first week and one session in the second week. Most 
of the TAs did say that some pupils had missed a few sessions because of staff or pupil absence, 
pupils forgetting to turn up to sessions, or scheduled trips or events. However, this was seen to be 
something that happened infrequently. 

The results of the two TA questionnaires showed that TAs were not consistently delivering two 
sessions per week for each pupil at the mid-point, but this had changed to almost all delivering two 
sessions per week by the end of the intervention. At the mid-point, 15 of the 22 respondents were 
delivering two sessions per week (568%) compared to 24 of the 27 TAs who returned the 
questionnaire at the end of the trial (89%). At the mid-point, five TAs were delivering one session per 
week (23%) and four were delivering four or more sessions per week (18%). By the end of the 
intervention, none of the TAs who responded to the questionnaire were delivering just one session 
per week, while three (1 1 %) had run three sessions per week for each pupil and one TA said that they 
ran either one or three sessions per week. Data provided by Catch Up on the number of sessions 
completed by each pupil indicates that each pupil received, on average, 32 sessions. An on-treatment 
analysis is described in the primary analysis section. 
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The two questionnaires revealed that TAs usually delivered all the sessions one to one (21 of 22 TAs 
at the mid-point, 96% and 26 of 27 TAs at the end of the intervention, 96%). The majority of TAs 
stated that they did not give Catch Up pupils literacy homework in addition to that given in their normal 
literacy classes (16 TAs (73%) at the mid-point survey and 14 TAs (52%) at the end of the 
intervention). The intervention itself did not contain a homework component but questions were asked 
to determine if treatment pupils were getting additional homework over and above what was being set 
as normal. Some TAs said they sometimes did set homework. 

When asked if the sessions usually lasted 15 minutes, most interviewees said that they did. However, 
several noted that during early sessions, when TAs were getting used to delivery, sessions tended to 
last longer than 15 minutes. This is in line with the responses seen in the TA questionnaires when at 
the mid-point of the intervention, 10 of the 22 TAs who responded (46%) said that the sessions with 
pupils usually lasted 15 minutes, while 7 TAs said that their sessions lasted 20 minutes (32%) and 2 
said their sessions lasted 30 minutes (9%). By the end of the intervention, 18 of the TAs who 
responded to the survey said that they spent 15 minutes delivering the session to pupils (67%) while 7 
said that they spent 20 minutes with the pupils (26%). 

At the mid-point of the intervention, 6 of the 22 TAs (27%) stated that they had adapted the 
intervention to some extent, while the rest of the TAs had not adapted it at all. At the end of the 
intervention, over half of the respondents had adapted the intervention; 10 of the 27 TAs (37%) had 
adapted the intervention to some extent, while 2 TAs had adapted it to a great extent and 3 had 
adapted it entirely. Twelve had not adapted it at all (44%). However, the interviewees did not appear 
to have adapted the intervention to any great extent, although a small number had increased the 
session to 20 minutes to include 5 minutes at the beginning of the session for general introductions 
and time for the pupil to settle in. 

The responses to both questionnaires revealed that TAs have been discussing the intervention with 
their colleagues. Of those responding to the mid-point questionnaire, 15 of the 22 respondents (68%) 
said that they had discussed their experiences with primary school colleagues either a little or a lot, 
while 17 of the 22 respondents (77%) had discussed their experiences with secondary school 
colleagues either a little or a lot. By the end of the intervention, sharing had increased with 21 of the 
27 TAs who responded to the end of intervention survey (78%) stating that they had shared their 
experiences with other TAs a little while 5 (19%) said that they had shared their experiences a lot. 

Outcomes 

Staff delivering and coordinating Catch Up Literacy in their schools were overwhelmingly positive 
about the impact of the programme. They drew upon a range of evaluative methods that they felt 
demonstrated solid programme impact. These methods included tracking tools provided by Catch Up 
Literacy, in-school reading tests, pupil feedback, observations, and more general anecdotal evidence. 
This section reports these outcomes including the results of the follow-up TA survey. 

Interviewees had observed the following programme impacts (note that these were self-reported in 
response to an open-ended interview question about impacts): 

• Half of interviewees reported progress in pupils’ general reading skills. 

• A third specified an increase in pupils’ reading age (by as much as 24 months for some 
pupils). 

• Around a third detailed improvements in pupils’ spelling skills. 

• Around a quarter felt that comprehension and understanding of texts was getting better. 

• Survey results show that 19 of the 27 TAs (70%) believed that Catch Up Literacy has 
improved pupils’ attainment in literacy to some or a great extent. 
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Added to these outcomes, over half of the staff said that they had seen a rise in the overall confidence 
of pupils as a result of Catch Up Literacy. Survey results support this finding as 21 of the 27 TAs (78 
%) thought that Catch Up Literacy has improved pupils’ confidence in literacy either entirely or to a 
great extent. Similarly, some staff felt that pupils’ self-esteem has improved, leading to increased 
levels of participation in other lessons. 

Staff also felt that after the intervention, some pupils were more engaged in reading, enjoyed it, 
actively chose to read in their spare time, and completed books they started reading in the Catch Up 
sessions. Supporting this finding, the survey demonstrates that 26 of the 27 TAs (96%) thought that 
the programme had improved pupils’ enjoyment of literacy to some or a great extent. 

Elements of the programme which contribute to its perceived success were identified as: 

• A one-to-one teaching approach. This enabled the TAs to build a relationship of trust, adapt 
the session to the needs of the individual, and work at an appropriate pace. In contrast to the 
classroom context, TAs felt that the pupils could ask questions and make mistakes without 
the risk of embarrassment. It also offered opportunities for praise and to listen to the pupil’s 
own individual concerns and opinions which contributed to raising self-esteem and a sense of 
self-worth. According to staff, this resulted in enjoyment of the sessions and a more positive 
attitude to attending. 

• Short sessions consisting of short activities. One coordinator described this as ‘covering the 
whole aspect of literacy in 15 minutes: reading, spelling, comprehension and writing.’ Added 
to this, short and focused activities reduced the need for pupils to ‘struggle through’ a full text 
or lesson. 

• Tailored sessions and resources. Tools provided by Catch Up Literacy enabled schools to 
attribute reading levels to books and other resources which ensured that texts were tailored to 
individual learning needs. 

• Reading strategies incorporated into the programme. Strategies such as ‘de-coding’ or 
identifying difficult words at the start of a chapter were considered effective by staff and they 
have also observed pupils using them in their lessons. 

In line with the programme design, interviewees generally felt that the Catch Up Literacy programme 
did not suit the ‘very weakest’ readers but was best suited to pupils who found it difficult to read ‘for 
meaning’ or struggled with inference and deduction. 

Teaching assistants were also very positive about the impact that Catch Up Literacy had on them as 
professionals. The ability to monitor and track pupil progress through the programme also enabled 
TAs to measure the impact of their own work. They, and a number of programme coordinators, 
described this as fundamental to focusing, tracking, and improving their own performance as 
professionals. In turn, this is said to give TAs confidence and contribute to their enjoyment of the role. 

The survey findings showed that: 

• 21 of the 27 TAs (78%) said that Catch Up Literacy has improved their confidence in 
supporting pupils who struggle with literacy either entirely or to a great extent, 

• 21 of the 27 TAs (78%) felt that Catch Up Literacy has improved their knowledge and skills in 
delivering literacy support to pupils entirely or to a great extent. 

• 19 of the 27 TAs (70%) said that delivering the Catch Up Literacy intervention has impacted 
upon their job satisfaction either entirely or to a great extent. 

Teaching assistants explained that they were now more analytical about their approach to teaching, 
considered new methods, and had also gained a different perspective on how pupils learn. For those 
new to the profession, the training and programme structure was a useful opportunity to gain a 
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grounding in literacy teaching and learning methods, while involvement in a formal Catch Up Literacy 
team was seen to support, reassure, and create a more professional status for the TAs. 

In some instances, coordinators explained that Catch Up Literacy had impacted more widely on 
school life by raising the profile of literacy interventions within delivery schools and led to further 
investment in future delivery, resources, and staffing. Furthermore, the programme design was 
reported to be an effective means of supporting relationship building between primary and secondary 
schools, staff and pupils. 

Staff were generally positive about their increased knowledge and engagement with research 
following involvement in the trial. In particular, one coordinator said that, although their school 
undertakes a fair amount of research already, staff now appreciated the value and benefits of 
undertaking an RCT despite their reluctance to carry them out in the past (due to ethical concerns). 
Other interviewees had appreciated the opportunity it gave them to reflect on their practice or to 
engage with research, made possible by immersion in a research project. In one school TAs reported 
that they have now started reading more widely about evidence-based practice while another said 
that they would continue to use the EEF toolkit to identify effective interventions. 

Formative findings 

A number of staff said that they would advise other schools to deliver Catch Up Literacy because they 
felt that it worked and was a ‘common sense approach to literacy education’. Success drivers 
included: identifying appropriate levels of staffing; adequate space for delivery; and additional time for 
planning, assessing pupils, and completing programme paperwork. Adequate time for the session 
itself should be timetabled in, with a realistic idea of how many pupils can be taught in a one-hour slot. 
Time should also be allocated to recruiting primary schools, relationship building, and sourcing pupil 
data. This helps to ensure the ‘right’ pupils are selected for the programme. 

Staff believed that their schools would make few changes to the way the programme was 
implemented in the future, and if anything, might recruit more staff to serve a larger cohort of pupils. 
However, they did identify the following potential improvements to the Catch Up Literacy intervention 
itself: 

• Provide teaching assistants with direct access to the online resources (rather than having to 
go through their coordinator). Catch Up protocols are for each TA to have their own log-in 
details. In practice this did not occur in every case. 

• Have more graded books readily available. One interviewee suggested that Catch Up Literacy 
could build links with a wider selection of publishers in order to achieve this. 

• Increase sessions to 20 minutes in order to allow more time for relationship building. 

• Provide clearer guidance on pupil ‘exit strategies’ from the programme. 

• Assign a reading ‘age’ to each book as well as a reading level. Some books have or may 
contain inappropriate content for the reader. 
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Conclusion 

Limitations 

The main limitation of the study is with the level of attrition between randomisation and the post-test in 
April. Although potentially a high attrition rate of 5.6%, analysis does show that there was no 
significant difference, on the pre-test measure, between those who dropped out of the study and 
those who remained within the ITT cohort. Another limitation is the potential introduction of bias when 
selecting pupils from the reserve list to join either the treatment group or the control. While analysis 
shows this group to be very small in number (only three of those included in the ITT analysis came 
from this list), there is potential for a biased selection of pupils. The main analysis was rerun excluding 
these three pupils and there is no reportable difference in model coefficients. Other than this relatively 
small limitation, the collection of eligible pupils and the randomisation process has resulted in a well- 
structured trial. 

The training and support received by teaching assistants resulted in a high degree of confidence in 
the delivery of the intervention. While both the interviews and questionnaires identified that teaching 
assistants were more confident by the end of the trial and had adapted the intervention to meet local 
needs, this would appear to be well within acceptable levels to consider the intervention has been 
delivered as the protocol and guidelines required. 

One other potential area for consideration as a limitation is with the intervention itself. As an 
intervention it is not a transition intervention, i.e. it is not specifically designed to address issues 
around a dip in performance over the period of transition between primary and secondary school. It is 
a programme that is designed to take place over a number of weeks with pupils receiving a number of 
one-to-one sessions. A period of time where a pupil does not receive the intervention is not part of the 
intervention’s normal delivery. Analysis of data at the start of Year 7 would suggest that any negative 
effects of the transition between primary and secondary impacted equally on treatment and control 
group pupils. To assess this, NGRT tests were administered at the start of Year 7 and a non- 
significant effect of 0.02 was identified between pupils in the intervention and control groups. 

Testing was organised and managed by the delivery organisation with TAs being provided with clear 
and detailed guidelines on test administration. While there was no independent monitoring of this 
process (although the evaluator did see the guidance document) there is no evidence to suggest that 
the administration of the tests, or their marking, were not undertaken following accepted protocols. 
Test administration and marking for both the pre and post-tests were, so the external evaluator was 
informed, carried out blind to group membership, with group membership being added to results after 
the marking had been undertaken. There is no evidence to suggest that this was not the case. 

Interpretation 

The primary analysis for this evaluation was to determine the impact of the Catch Up Literacy 
evaluation to increase literacy scores. The primary analysis identifies, and based on this trial, that 
there is insufficient evidence to reject the null hypothesis of there being no difference between the 
treatment and control groups, although the effect size is small. The prior attainment of pupils was 
controlled for in the models by the pupils’ reading score at Key Stage 2. Additional models were run 
looking at effects for pupils eligible for free schools meals, pupil gender, and age. The overall 
interpretation of these models is that there is insufficient evidence to claim that the Catch Up Literacy 
intervention is having an effect over and above normal classroom activities. One-to-one teaching has 
been shown to have a number benefits for increasing pupil abilities in literacy and numeracy. The 
recent trial looking at Catch Up Numeracy (Rutt et al., 2014) identified a significant effect of one-to- 
one teaching conducted by TAs and the EEF toolkit identifies a number of trials that have shown the 
significant effects of this type of delivery. This trial, while not identifying any gains in literacy scores, 
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has identified a couple of significant gains in pupils’ attitude to school and in their reading confidence. 
These are very relevant findings and, while considered as part of the secondary analyses, should be 
seen as being highly beneficial, and necessary, when wishing to improve pupil levels of literacy. 
Attitudes and confidence may need to be improved first before any improvements in test scores can 
be observed. 

It is evident from the process evaluation, and the questionnaires, that TAs and Catch Up coordinators 
valued the intervention and believed it had a positive impact on some pupils. These impacts were felt 
to be on literacy outcomes as well as around softer outcomes such as confidence, engagement with 
learning, and attitude to school, although more improvements were identified for the softer outcomes. 
As for TAs, outcomes from the telephone interviews and the questionnaires clearly identified the 
benefits they felt the Catch Up Literacy intervention had had on their own satisfaction and on their 
personal and professional development. More engaged staff can only be a good thing for a school 
and its pupils. 

It was also clear from the process evaluation that delivering a structured intervention within a school 
can be problematic due to pupil absences, TAs’ timetables, and conflicting demands on TAs’ time. An 
intervention such as Catch Up needs to be planned into the timetable from the outset so that 
resources can be allocated, particularly to identify an enclosed and private space due to the need for 
reading out loud. 
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Appendix A 



Evaluation of Catch Up 8 Literacy - 
Student Questionnaire 


We are doing some research on Catch Up* Literacy. We would like you to answer 
some questions so we can find out what you think about reading and writing, school 
and yourself. The questions will take you about 1 5 minutes to answer. 

All your answers to these questions will be kept private and we will not use your 
name in any of our reports. Please use BLACK INK to complete this questionnaire. 


About you 


First Name: 


Last Name: 


Date of Birth: 

drnnrnm 






1. How much do you agree with the following statements: 

(Please tick one box on each row) 



Strongly 
a yee 

i 

Agree 

2 

Not sue 
3 

Disagree 

4 

Strongly 

disagree 

5 

Did not 
do this 
e 

1 enjoy reading 

□ 

□ 

□ 

□ 

□ 


1 feel confident reading out loud to 
the dass 

□ 

□ 

□ 

□ 

□ 


1 know how to deal with words 1 find 
hard to read 

□ 

□ 

□ 

□ 

□ 


1 enjoy reading in my own time 

□ 

□ 

□ 

□ 

□ 


Reading is one of my favourite 
things to do 

□ 

□ 

□ 

□ 

□ 


1 enjoy talking to my friends/ family 
about the books 1 have read 

□ 

□ 

□ 

□ 

□ 


1 enjoy writing 

□ 

□ 

□ 

□ 

□ 


1 find writing easy 

□ 

□ 

□ 

□ 

□ 


1 feel confident about writing in 
sentences 

□ 

□ 

□ 

□ 

□ 


Writing is one of my favourite things 
to do 

□ 

□ 

□ 

□ 

□ 


1 am doing well in reading 

□ 

□ 

□ 

□ 

□ 


1 am doing well in writing 

□ 

□ 

□ 

□ 

□ 


1 liked working with the same 
Teaching Assistant dicing Year 0 & 
Year 7 for Catch Up # Literacy 

□ 

□ 

□ 

□ 

□ 

□ 

■ 


i 




1 
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2. How much do you agree with the following statements: 


(Please tick one box on each row) 



Strongly 

agree 

Agree 

Not sure 

Disagree 

Strongly 

disagree 


i 

2 

3 

4 

5 

It Is important to do well at school 

□ 

□ 

□ 

□ 

□ 

1 try hard at school 

□ 

□ 

□ 

□ 

□ 

Most of the time 1 feel confident 
about doing my work in lessons 

□ 

□ 

□ 

□ 

□ 

Most of the time 1 enjoy school 

□ 

□ 

□ 

□ 

□ 

1 behave well in lessons 

□ 

□ 

□ 

□ 

□ 

1 always do my homework 

□ 

□ 

□ 

□ 

□ 


3. How much do you agree with the following statements: 


(Please tick one box on each row] 



Strongly 

agree 

i 

Agree 

2 

Not sure 
3 

Disagree 

4 

Strongly 

disagree 

5 

1 feel happy 

□ 

□ 

□ 

□ 

□ 

1 am a confident person 

□ 

□ 

□ 

□ 

□ 

1 have people to talk to if 1 feel 
unhappy or worried 

□ 

□ 

□ 

□ 

□ 

1 like the way 1 am 

□ 

□ 

□ 

□ 

□ 

1 worry when 1 have to meet new 
people 

□ 

□ 

□ 

□ 

□ 

1 feel positive most of the time 

□ 

□ 

□ 

□ 

□ 


Thank you very much for answering our questions. Please put this 
questionnaire into the envelope and hand it to your teacher. 
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Evaluation of Catch Up® Literacy - 
Teaching Assistant Survey 


As you are aware, the National Foundation for Educational Research (NFER) is currently 
undertaking an evaluation of Catch Up Literacy. As a Teaching Assistant (TA) participating 
in the project, we would appreciate it if you could complete this short survey. 

Our research has been commissioned by the Education Endowment Foundation (EEF), an 
independent charity dedicated to raising the attainment of disadvantaged pupils by 
challenging educational disadvantage, sharing evidence and finding out what works. 

The purpose of this survey is to explore your views and experiences of delivering 
Catch Up Literacy to year 6 pupils (i.e. the questions do not relate to any other literacy 
support you provide to other pupils not part of this project). This survey should take no 
more than ten minutes to complete. We would be grateful if you could complete the 
questionnaire in BLACK INK by 19th July 2013 and return it directly to NFER in the pre- 
paid envelope provided. 

Please be assured that your answers will be treated in confidence by the NFER research 
team. Survey data will only be shared with EEF in anonymised form. It will be reported in 
aggregated form together with everyone else’s responses so individual responses will not 
be shared with Catch Up, nor the primary or secondary schools within which you will work 
as part of this project. 

If you have any queries please do not hesitate to contact Helen Francis on 01753 637 344 
or email h.francis@nfer.ac.uk 

Thank you in advance for your help with this survey. 


Your Catch Up Literacy sessions 


1. Typically, how many Catch Up Literacy sessions have you held with each project 
pupil per week (including your assessment sessions)? (Please tick one box) 


None Q ; 


One Two Q 3 Three [J 4 Fo “ r Q ° e r Qs 


2. Roughly, how long do you spend per session : 
(Please enter whole numbers only or tick not sure) 


Preparing resources/reading materials 
for the assessments ? 

Preparing resources/reading materials 
for the intervention? 


With your pupil in the sessions? 


Minutes 

Not sure 

□ 

Minutes 

Not sure 

□ 

Minutes 

Not sure 

□ 


Education Endowment Foundation 


34 







Catch Up Literacy 


3. For each person listed below, please indicate the extent to which they set the work 
for your Catch Up Literacy sessions: 

(Please tick one box on each row) 



Not at all 

Rarely 

Sometimes 

Usually 

Don’t know 


i 

2 

3 

4 

5 

Yourself 

□ 

□ 

□ 

□ 

□ 

Year 6 class literacy 
teacher 

□ 

□ 

□ 

□ 

□ 

Primary school literacy co- 
ordinator 

□ 

□ 

□ 

□ 

□ 

Secondary school literacy 
co-ordinator 

□ 

□ 

□ 

□ 

□ 

Your in school 
Catch Up Literacy 
co-ordinator 

□ 

□ 

□ 

□ 

□ 

Your Catch Up Literacy 
trainer 

□ 

n 

□ 

□ 

□ 

Someone else (please specify 
below): 

□ 

□ 

□ 

□ 

□ 


4. Please answer the following questions thinking about your Catch Up Literacy 
sessions: (Please tick one box on each row) 


To what extent are your... 


Not at all 

i 


Rarely 

2 


Sometimes 

3 


Usually 

4 


Don't know 

5 


Catch Up Literacy sessions delivered 
one-to-one? 


□ 


□ 


□ 


□ □ 


sessions taking place at the same time i — i 

as the pupils' normal literacy lessons? ' — ' 

Catch Up Literacy project pupils given 
any literacy homework (including 
Catch Up digital games) in addition to 
what is issued in their normal literacy 

classes? 


□ 

□ 


□ 

□ 


□ □ 

□ □ 
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5. To what extent have you shared your experiences of Catch Up Literacy with other 
TAs (not involved in Catch Up Literacy): 

(Please tick one box on each row only) 



Not at all 

A little 

A lot 

Not sure 


i 

2 

3 

4 

At the primary school 

□ 

□ 

□ 

□ 

At the secondary school 

□ 

□ 

□ 

□ 


Catch Up Literacy intervention 


6. The following questions ask you about the Catch Up Literacy intervention 
( excluding the assessment). (Please tick one box on each row) 



Not at all 

To some 
extent 

To a great 
extent 

o 

Entirely 

Don't 

know 

c 

To what extent do you feel 
Catch Up Literacy differs from normal 
primary school literacy practice? 

□ 

Z 

□ 

J 

□ 

□ 

O 

□ 

To what extent do you feel 
Catch Up Literacy differs from normal 
secondary school literacy practice? 

□ 

□ 

□ 

□ 

□ 

How well equipped do you currently 
feel to deliver Catch Up Literacy? 

□ 

□ 

□ 

□ 

□ 

To what extent have you adapted the 
Catch Up Literacy intervention? 

□ 

□ 

□ 

□ 

□ 


7. Have you ever received Catch Up Numeracy training? (Please tick one box only) 


Yes No Not sure 

□ □ □ 
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Background information 

8. Please indicate whether you are male or female? (Please tick one box only) 


Male Female Prefer not to say 

□ □ □ 


9. Which age bracket do you fall into? (Please tick one box only) 


16-24 

□ ’ 

55-64 D 5 

25-34 

□ 2 

65 and above | 6 

35-44 

□ 3 

Prefer not to say 7 

45-54 



10. How long have you worked as a teaching assistant (either in this school, or 

others)? (Please tick one box only) 


Less than 1 year 



1-5 years 

□ 2 


6-10 years 

□ 3 


1 1 years + 



Don’t know 

□ 5 



11. Have you previously trained and/or worked as a classroom teacher? 

(Please tick one box only) 


Yes No 

□ □ 


12. What is the highest level of qualification you have gained? 

(Please tick one box only) 


No qualifications 

O 

First degree 

□ 

GCSE/O Level/CSE 

□ 2 

Higher degree 

□' 

NVQ Level 3 

□ 3 

Don’t know 

□ 

A Levels/BTEC/IB 


Other (please specify below): 



Thank you for completing the questionnaire. 

Please return this questionnaire, using the pre-paid envelope, to: 
Research Operations, The NFER, The Mere, Upton Park, Slough, SL1 2DQ. 

■ ■ 
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Evaluation of Catch Up® Literacy - 
Teaching Assistant Survey 


As you are aware, the National Foundation for Educational Research (NFER) is currently 
undertaking an evaluation of the Catch Up* Literacy intervention (“the project"). To help us 
with our research, as a Teaching Assistant (TA) participating in the project, we would 
appreciate it if you could complete this short survey. 

Our research has been commissioned by the Education Endowment Foundation (EEF), an 
independent charity dedicated to raising the attainment of disadvantaged pupils by 
challenging educational disadvantage, sharing evidence and finding out what works. 

This survey focuses on you and your experiences delivering Catch Up Literacy (i.e. it does 
not relate to any literacy support you provide to other pupils not part of this project). It 
should take no more than ten minutes to complete. We would be grateful if you could 
complete the questionnaire in BLACK INK and please return it by 28 th March 2014 directly 
to NFER in the pre-paid envelope provided. 

Please be assured that your answers will be treated in confidence by the NFER research 
team, and individual responses will not be shared with Catch Up or your school. 

If you have any queries, please do not hesitate to contact Keren Beddow on 01753 637 338 
or email k.beddow@nfer.ac.uk 

Thank you in advance for your help with this survey. 


Your Catch Up Literacy sessions 


When answering these questions, please consider your practice when delivering Catch Up . 


1. Typically, how many Catch Up Literacy sessions did you hold with each project 
pupil per week? (Please tick one box) 


None □> °ne □ , Two Qj Three □ < F ° n U I?I □ . 


2. Roughly, how long did you spend per session: 

(Please enter whole numbers only or tick not sure) 


Preparing resources/ activities/ 
materials beforehand? 


With your pupil in the sessions? 



Minutes 


Minutes 


Not sure [ 
Not sure | 
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3. For each person listed below, please indicate the extent to which they set the work 
for your Catch Up Literacy sessions: (Please tick one box on each row) 

Yourself 

Not at all 

i 

□ 

Rarely 

2 

□ 

Sometimes 

3 

□ 

Usually 

4 

□ 

Don’t know 

5 

□ 

Class literacy teacher 

n 

n 

□ 

□ 

□ 

Literacy co-ordinator 

□ 

□ 

□ 

□ 

□ 

Catch Up co-ordinator 

□ 

□ 

□ 

□ 

□ 

Someone else 

□ 

□ 

□ 

□ 

□ 

4. Please answer the following questions thinking about your Catch Up Literacy 
sessions (Please tick one box on each row) 

To what extent were your... 

Not at all 

i 

Rarely 

2 

Sometimes 

3 

Usually 

4 

Don’t know 

5 

sessions delivered one-to-one? 

□ 

□ 

□ 

□ 

□ 

sessions taking place at the 
same time as the pupils’ normal 
literacy lessons? 

□ 

□ 

□ 

□ 

□ 

Catch Up Literacy project pupils 
given any literacy homework in 
addition to what was issued in 
their normal literacy classes? 

□ 

□ 

□ 

□ 

□ 

5. To what extent have you shared your experiences of Catch Up Literacy with other 
TAs at your school? (Please tick one box only) 

Not at all A little 


A lot 

Not sure 


1 2 


3 


4 


□ 

□ 


□ 


□ 


6. The following questions ask you about the Catch Up Literacy intervention 

(Please tick one box on each row) 



Not at all 

To some 
extent 

To a great 
extent 

Entirely 

Don't know 

To what extent do you feel 

i 

□ 

2 

3 

4 

5 

Catch Up Literacy differs from 
normal TA practice? 

How well equipped do you 

□ 

□ 

□ 

□ 

currently feel to deliver 
Catch Up Literacy? 

To what extent did you adapt 

□ 

□ 

□ 

□ 

□ 

the Catch Up Literacy 
intervention? 

□ 

□ 

□ 

□ 

□ 
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Impact of the Catch Up Literacy intervention 


7. The following questions ask about your perceptions of the impact of the Catch Up 
Literacy intervention on the pupils you taught as part of this project 

(Please tick one box on each row) 


Not at all 

To some 

To a great 

Entirely 

Don’t know 

To what extent do you feel... 


extent 

extent 




1 

2 

3 

4 

5 

Catch Up Literacy has improved 
pupils’ enjoyment of literacy? 

□ 

□ 

□ 

□ 

□ 

Catch Up Literacy has improved 
pupils’ confidence in literacy? 

□ 

□ 

□ 

□ 

□ 

Catch Up Literacy has improved 
pupils’ attainment in literacy? 

□ 

□ 

□ 

□ 

□ 

8. The following questions ask about your perceptions of the impact of the Catch Up 

Literacy intervention on your role as a TA (Please tick one box on each row) 



Not at all 

To some 

To a great 

Entirely 

Don't know 

To what extent... 


extent 

extent 




1 

2 

3 

4 

5 

has Catch Up Literacy improved 

□ 

□ 

□ 

□ 

□ 

your confidence in supporting pupils 

who are struggling with literacy? 






has Catch Up Literacy improved 

□ 

□ 

□ 

□ 

□ 

your knowledge and skills in 

delivering literacy support to pupils? 






has delivering the Catch Up Literacy 
intervention impacted upon your job 

□ 

□ 

□ 

□ 

□ 

satisfaction? 







9. Have you ever received Catch Up Numeracy training? (Please tick one box only) 


Yes 

No 

Not sure 

□ 

□ 

□ 
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Background information 


10. Please indicate whether you are male or female? (Please tick one box only) 


Male Female Prefer not to say 

□ □ □ 


1 1 . Which of the following age brackets do you fall into? (Please tick one box only) 


16-24 

□ ’ 

55-64 

□ 

25-34 

□ 2 

65 and above 

□ 

35-44 

□ * 

Prefer not to say 

□ 

45-54 

□ < 




12. How long have you worked as a teaching assistant (either in this school, or 
Others)? (Please tick one box only) 


Less than 1 year 

□ ’ 

1 1 years + 

1-5 years 

□ 2 

Don’t know 

6-10 years 

□ 3 



13. Have you previously trained and/or worked as a classroom teacher? 

(Please tick one box only) 


Yes No 

□ □ 


14. What is the highest level of qualification you have gained? 

(Please tick one box only) 


No qualifications 


First degree 

□ 

GCSE/'O Level/CSE 

□ 2 

Higher degree 

□ 

NVQ Level 3 

□ 2 

Don’t know 

□ 

A Levels/BTEC/IB 


Other (please specify below): 



Thank you for completing this questionnaire. 

Please return this questionnaire, using the pre-paid envelope, to: 
Research & Product Operations, NFER, The Mere. Upton Park, 
Slough, SL1 2DQ 
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Appendix D 

Consent Letter 

[Primary school headed paper] 


Dear Parent/Carer 

THE CATCH UP LITERACY - Year 6 and Year 7 research project 


(name of school) has been selected by Catch Up , a not-for-profit, registered 

charity, to take part in a research project to look at the benefits of Catch Up Literacy for Year 6 and Year 7 pupils. 
The project will start in April 2013 and will involve only 17 secondary schools across England, with some Year 6 
pupils in some of these secondary schools’ linked/feeder primary schools. The project will continue up to April 
2014. 

During this period some pupils will be randomly selected by the project to receive Catch Up Literacy sessions. 
This will start in Year 6 and continue in Year 7. The random selecting will be undertaken by the National 
Foundation for Educational Research (NFER) (an educational research organisation) and will take place after the 
project starts in April 2013. All the pupils will take part in testing in Year 7. 

All the data will be used in an anonymised way and your child or the school will not be identified in any reports 
about the research. Your daughter/son cannot take part in the project without your permission and we would like 
to ask that you complete and return all this form to give permission for your child to take part in this exciting 
project. 

You can choose to withdraw your child from the project at a later point and if you would like any further 
information please let me know. 

Yours sincerely 

Name of school contact 

I (name of parent or guardian) give permission for 

(name of child) from 

(name of school) to take part in the Catch Up Literacy Year 6 and Year 7 project. 


Signed 


Date 
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Appendix C: Padlock rating 



2. Power : 
What is 
minimum 
detectable 
effect at the 
start? 



Y 

Evaluation design 


J V. 


Implementation 


J V_ 


5. Threats to 


validi 


Adjustment to 
account for issues 
with 

interpretation? 


Y 

Analysis and interpretation 


r 

j 



The final security rating for this trial is 4 i . This means that the conclusions have moderate to high 
security. 

The trial was designed as an efficacy trial and could achieve a maximum of 5 6 . This was a well- 
designed effectiveness trial. The MDES was above 0.2, and below 0.3, reducing the padlocks to 4. 
The level of attrition was relatively low (8%), so the padlocks remained at 4. There was evidence of 
an imbalance at baseline in prior attainment of 0.11, which would have resulted in a loss of 2 padlocks 
had there not been evidence that this was due to chance and not to attrition. Instead, only 1 was lost. 
Finally, since the intervention was delivered by TAs, the fact that the attainment measures were 
delivered by TAs is a threat to the validity. Therefore, the final padlock rating is 4. 
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Appendix D: Cost rating 


Cost ratings are based on the approximate cost per pupil of implementing the intervention over one 
year. Cost ratings are awarded using the following criteria. 


Cost 

Description 

£ 

Very low: less than £80 per pupil per year. 

£ £ 

Low: up to about £170 per pupil per year. 

£££ 

Moderate: up to about £700 per pupil per year. 

£ £ £ £ 

High: up to £1 ,200 per pupil per year. 

£ £ £ £ £ 

Very high: over £1 ,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 

To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 

Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 

This document is available for download at www.educationendowmentfoundation.org.uk 
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